AWS SSM Agent – Connection Error

When trying to access EC2 instance using AWS ssm cli or SSM connect manager and get error Plugin with name Standard_Stream not found. Step name: Standard_Stream. No worry, this post shows you how to trouble shoot

What’s In This Do…


This content originally appeared on DEV Community and was authored by Vu Dao

When trying to access EC2 instance using AWS ssm cli or SSM connect manager and get error Plugin with name Standard_Stream not found. Step name: Standard_Stream. No worry, this post shows you how to trouble shoot

What’s In This Document

? What is the error

  • We got error when trying access EC2 instance using SSM agent and AWS CLI 
⚡ $ aws ssm start-session --target i-123abc456efd789xx --region ap-northeast-2

Starting session with SessionId: userdev-004f77465f262084d


SessionId: userdev-004f77465f262084d : Plugin with name Standard_Stream not found. Step name: Standard_Stream
  • Event from console

Alt-text

? Investigate and Apply solution

  • So far you need to access the EC2 using SSH with key-pem to debug (ask you admin)

    • Running tail -f got issue

      tail: inotify resources exhausted
tail: inotify cannot be used, reverting to polling

    • Restart ssm-agent service also got issue No space left on device but it's not about disk space

      [root@env-test ec2-user]# systemctl restart amazon-ssm-agent.service
Error: No space left on device

[root@env-test ec2-user]# df -h |grep dev
devtmpfs         32G     0   32G   0% /dev
tmpfs            32G     0   32G   0% /dev/shm
/dev/nvme0n1p1  100G   82G   18G  83% /

  • So the error itself means that system is getting low on inotify watches, that enable programs to monitor file/dirs changes. To see the currently set limit (including output on my machine)

⚡ $ cat /proc/sys/fs/inotify/max_user_watches
8192
  • Check which processes using inotify to improve your apps or increase max_user_watches 
# for foo in /proc/*/fd/*; do readlink -f $foo; done | grep inotify | sort | uniq -c | sort -nr
      5 /proc/1/fd/anon_inode:inotify
      2 /proc/7126/fd/anon_inode:inotify
      2 /proc/5130/fd/anon_inode:inotify
      1 /proc/4497/fd/anon_inode:inotify
      1 /proc/4437/fd/anon_inode:inotify
      1 /proc/4151/fd/anon_inode:inotify
      1 /proc/4147/fd/anon_inode:inotify
      1 /proc/4028/fd/anon_inode:inotify
      1 /proc/3913/fd/anon_inode:inotify
      1 /proc/3841/fd/anon_inode:inotify
      1 /proc/31146/fd/anon_inode:inotify
      1 /proc/2829/fd/anon_inode:inotify
      1 /proc/21259/fd/anon_inode:inotify
      1 /proc/1934/fd/anon_inode:inotify
  • Notice that the above inotify list include PID of ssm-agent processes, it explains why we got issue with SSM when max_user_watches reached limit 
# ps -ef |grep ssm-ag
root      3841     1  0 00:02 ?        00:00:05 /usr/bin/amazon-ssm-agent
root      4497  3841  0 00:02 ?        00:00:33 /usr/bin/ssm-agent-worker

- Final Solution: Permanent solution (preserved across restarts)

echo "fs.inotify.max_user_watches=1048576" >> /etc/sysctl.conf
sysctl -p
  • Verify: 
⚡ $ aws ssm start-session --target i-123abc456efd789xx --region ap-northeast-2

Starting session with SessionId: userdev-03ccb1a04a6345bf5
sh-4.2$

? Conclusion

  • This issue comes from EC2 instance not about SSM agent
  • Go to ? to undestanding SSM agent in 2 minutes.

? Blog · Github · stackoverflow · Linkedin · Group · Page · Twitter ?


This content originally appeared on DEV Community and was authored by Vu Dao


Print Share Comment Cite Upload Translate Updates
APA

Vu Dao | Sciencx (2021-06-25T11:49:21+00:00) AWS SSM Agent – Connection Error. Retrieved from https://www.scien.cx/2021/06/25/aws-ssm-agent-connection-error/

MLA
" » AWS SSM Agent – Connection Error." Vu Dao | Sciencx - Friday June 25, 2021, https://www.scien.cx/2021/06/25/aws-ssm-agent-connection-error/
HARVARD
Vu Dao | Sciencx Friday June 25, 2021 » AWS SSM Agent – Connection Error., viewed ,<https://www.scien.cx/2021/06/25/aws-ssm-agent-connection-error/>
VANCOUVER
Vu Dao | Sciencx - » AWS SSM Agent – Connection Error. [Internet]. [Accessed ]. Available from: https://www.scien.cx/2021/06/25/aws-ssm-agent-connection-error/
CHICAGO
" » AWS SSM Agent – Connection Error." Vu Dao | Sciencx - Accessed . https://www.scien.cx/2021/06/25/aws-ssm-agent-connection-error/
IEEE
" » AWS SSM Agent – Connection Error." Vu Dao | Sciencx [Online]. Available: https://www.scien.cx/2021/06/25/aws-ssm-agent-connection-error/. [Accessed: ]
rf:citation
» AWS SSM Agent – Connection Error | Vu Dao | Sciencx | https://www.scien.cx/2021/06/25/aws-ssm-agent-connection-error/ |

Please log in to upload a file.



There are no updates yet.
Click the Upload button above to add an update.

You must be logged in to translate posts. Please log in or register.

Related Posts