AWS Security: The Shared Responsibility Model
Most major public cloud providers including, Amazon Web Services (AWS), follow a “Shared Responsibility Model” for security and compliance. This means that security and compliance tasks are a shared responsibility between the cloud platform and the cloud customer. This applies for regulatory compliance such as HIPAA, PCI DSS, and FedRAMP, but also cybersecurity frameworks such as NIST, ISO, and SOC. AWS customers are able to take advantage of established AWS certifications and security programs to jump-start compliance efforts. Cloud providers implement certain physical security protections, but customers are responsible for building secure solutions with these cloud services.
Why Collect Security Events?
Most cybersecurity frameworks and regulatory standards have set requirements for organizations to implement solutions around audit logging, backups and disaster recovery (DR), vulnerability scanning, intrusion detection systems (IDS), and firewall/networking. Organizations may implement these solutions internally or turn to a number of different vendors.
Since managing large cloud environments and multiple security solutions can be difficult, organizations often turn to SIEM solutions logging solutions such as the ELK stack. The “ELK stack” is based on Elastic’s opensource solutions Elasticsearch, Logstash, Kibana. Organizations can deploy the ELK stack themselves or turn to hosted offerings of ELK such as Elastic Cloud, AWS Elasticsearch, and Logz.io. ELK can help organizations better view and manage overall security events and security operations. Using a log aggregation solution makes it easier for organizations to oversee security events, service availability, and potential suspicious activity.
What To Collect From AWS?
Amazon Web Services provides many different cloud services, from traditional virtual machines and data storage to serverless and container-based workloads. In order to gather insight into AWS cloud environments, organizations may consider using Logstash/ELK stack and AWS log information including:
VPC Flow Logs – AWS allows organizations to collect account activity (such as user logins) as well as region activity related to individual cloud services in the regions.
Permissions and IAM Logs – Organizations may monitor changes to permissions and events that occur related to Amazon Identity and Access Management (IAM).
Cloud Service Access Logs – Access logs can be collected from individual cloud services such as S3, RDS, and Redshift. This allows team to see individual service queries and access.
System Logs – Organizations using EC2 instances or E containers may collect logs operating system logs (syslog, etc).
Intrusion Detection Logs – Organizations may collect suspicious access attempts for cloud services from intrusion detection systems (IDS).
Vulnerability Scanning Logs – Organizations may collect information related to operating system vulnerabilities and manage this information in connection to an organization patching schedule.
Cloud Configuration Logs – Organizations may collect logs related to changes in overall cloud configuration, cloud resources being created, or general configuration management and orchestration.
Once this information is collected in Logstash, security teams can build reports and visualizations and evaluate the security and compliance state for the organization. Security teams may work with DevOps staff and other team members to resolve security issues.
Sending AWS Data To Logstash/ELK
There are several ways to connect Logstash and AWS. Teams may send AWS cloud service logs to Logstash and may configure system specific logging for EC2 instance and other systems. Cloud information can be aggregated and delivered to Logstash or other SIEM solutions through the following approaches:
- Teams may enable AWS Elasticsearch and start collecting logs by streaming Cloudwatch streams and other service streams right into AWS.
- Teams may deliver logs to hosted ELK/Logstash services via Amazon Cloudwatch and AWS Cloudtrail or a variety of other outputs. Organizations may consider structuring and collecting logs in Cloudwatch, and using AWS Lambda or other services to send logs over to hosted ELK services.
Gathering Cloud Compliance Events with Dash
Collecting cloud security events in ELK is a good step towards visualizing security issues, but this information, does not give organizations the full picture into their state of compliance. Security teams must gather, assess, and determine how events in their cloud environment affect compliance with regulatory standards such as HIPAA/HITECH, PCI DSS, FedRAMP, as well as cybersecurity frameworks such as the NIST CSF, ISO 27001 and SOC.
The Dash Compliance Automation Platform makes it easy for organizations to gather regulatory compliance events for AWS and the public cloud. Security teams can configure Dash security policies, gather compliance information, and stream compliant events to Logstash and other SIEMs. Companies can collect cloud security issues that are mapped to HIPAA, NIST CSF, and other regulatory standards and compliance frameworks. Security issues include issues such as:
Publicly Available S3 Buckets
HIPAA: 164.312(a)(1) – Access Control
NIST CSF: NIST SP 800-53, NIST SP 800-63, NIST SP 800-21, NIST SP 800-34, FIPS 140-2
ISO: 9.1.1, 9.4.1, 9.6.1, 12.1.3
Unencrypted EBS Volumes
HIPAA: 164.312(a)(2)(iv) – Encryption and Decryption
NIST CSF: NIST SP 800-63, NIST SP 800-21, NIST SP 800-34, FIPS 140-2, NIST SP 800-12
ISO: 8.5.1, 8.7.4, 10.3.1, 10.3.2, 10.3.3, 12.1.6
RDS cluster configured with only one availability zone (AZ)
HIPAA: 164.310(a)(2)(i) Contingency Operations
NIST CSF: NIST SP 800-18 Rev 1
ISO: 7.2.2, 11.1.1, 11.1.3, 12.1.3, 4.1.7, 7.2.3, 7.2.4, 8.1.1
IAM AssumeRole is misconfigured
HIPAA: 164.312(c)(2) Mechanism to Authenticate Electronic Protected Health Information,
164.312(d) Person or Entity Authentication
NIST CSF: NIST SP 800-14, NIST SP 800-53 Rev 4, NIST SP 800-106, NIST SP 800-107
ISO: 10.2.3, 8.1.6
With Dash, organizations can simplify their security operations and compliance management and have instant insight into their cloud security and compliance programs.
Maintaining AWS Security and Compliance With Logstash
After gathering AWS security and compliance information in Logstash, organizations must manage security operations (SecOps) over time. Security teams should develop processes for identifying and resolving security issues. Organizations may also turn to Dash for setting security standards and policies that fit the organization.
Determine types of issue – Determine what type of security events will require response and remediation.
Set security policies – Set policies for security and compliance standards that will be upheld by the security team (IE. disaster recovery policies, vulnerability scanning policies).
Create roles – Assign roles to staff related to assessing and resolving security and compliance events.
Create processes for creating security and DevOps tickets – Connect security events into organization workflow and ticketing, so teams can resolve security events.
Continuously assess security and compliance – Continuously view security events and respond to security events within your organization. Teams should access security policies on a frequent basis.
Following these steps will enable teams better streamline the security process. Teams should create policies that fit into their staff structure and technologies. Learn how Dash can be used in coordination with a logging solution or SIEM to manage security policies and conduct continuous compliance monitoring.
Add Dash Compliance Stream To Logstash
FAQ: Operating the ELK Stack for AWS Security
Is the ELK stack the same as Amazon OpenSearch Service?
Not exactly. Elasticsearch and Kibana were forked into OpenSearch after Elastic changed its licensing, and Amazon OpenSearch Service is the managed successor to the older Amazon Elasticsearch Service offering. The concepts—indexing, search, dashboards—carry over, but APIs, plugins, and some features differ. Verify compatibility before migrating existing pipelines or dashboards between them.
Should you self-host ELK or use a managed service?
Self-hosting gives full control over configuration, data location, and cost at scale, but you own cluster operations, upgrades, scaling, and availability. Managed offerings reduce operational burden and are usually faster to start, at a higher per-unit cost and with less control over the underlying cluster. The decision usually turns on team capacity and how much log volume you expect to retain.
Why does log ingestion cost so much, and how do you control it?
Costs are driven by volume ingested, storage retained, and query compute. Common controls include filtering or dropping low-value events at the collection layer, sampling verbose sources, setting index lifecycle policies that move older data to cheaper storage and delete it when retention ends, and avoiding indexing fields you never query. Review the largest contributors by source before optimizing broadly.
What is an index lifecycle policy, and why does it matter for compliance?
An index lifecycle policy automates how indices move through hot, warm, cold, and delete phases. For compliance, it enforces that logs are retained for the required period and then removed, rather than accumulating indefinitely or expiring prematurely. Document the retention period per log source and confirm the policy matches it.
Can PHI end up in Elasticsearch, and how should that be handled?
Yes. Application logs, error traces, and request payloads can contain identifiers or health details. Treat the cluster as a system that may hold ePHI: restrict access, encrypt data at rest and in transit, redact sensitive fields before indexing where possible, and apply retention limits. Include the cluster in your risk analysis and access reviews.
How do you secure the ELK stack itself?
The cluster often holds the most sensitive operational data in your environment, so protect it accordingly. Enable authentication and role-based access, use TLS for transport and the Kibana interface, keep the cluster off the public internet, and restrict network access to known sources. Apply patching to Elasticsearch, Logstash, and Kibana, and monitor administrative actions on the cluster.
Is ELK a SIEM, and does it satisfy detection requirements?
ELK is fundamentally a log aggregation and search platform. Detection requires additional layers: correlation rules, alerting, threat intelligence, and a defined triage process with owners. Many teams build these on top of ELK or pair it with a dedicated SIEM. Collecting logs alone does not demonstrate monitoring—what matters is that alerts are defined, routed, and acted upon.
How do you keep the pipeline from silently failing?
Monitor the pipeline as a production system: track ingestion rates per source, alert when expected log streams stop arriving, watch for parsing failures and dropped events, and monitor cluster health and disk usage. A common failure mode is a source that quietly stops sending while dashboards continue to look normal because they only show historical data.
