Team Atmecs

Cloudops Automation Project ATMECS – Content Team The client is the world’s leading independent, end-to-end IT services company, serving nearly 6,000 private and public sector clients from a diverse array of industries across 70 countries. The company’s technology independence, global talent and extensive partner network help clients harness the power of innovation to thrive on change and guide their digital transformation journeys. Challenges: Development of single pane of glass platform to monitor, automate, orchestrate multiple public and private cloud offerings.  To implement single design to support all cross offerings in the public and private cloud family. Cost optimization by transiting operation from DevOps to DevSecOps. Solution Approach: The client platform is a digital-generation delivery platform that underpins: Intelligence, which provides the client digital platform ecosystem with the ability to ingest and process a wide spectrum of data types  Orchestration, which focuses on the ability of the platform to organize, execute, and track the processes and workflows  Automation, which encompasses the technologies that enable the mechanization of tasks within the delivery environment services to managed service offerings for public and private cloud. ATMECS Contribution: CloudOps Dashboard Development Operational Automation for AWS and Azure  Azure Powershell automation  AWS Boto automation script  Vault Implementation for Securing customer credentials Setting up CI/CD process Cloudops Automation Project

Building IoT platform for an Ad network company ATMECS – Content Team Client is the next-generation technology for digital, out-of-home market. It creates interactive experience of digital marketing anywhere, on any space. Client’s solutions allow its customers to elevate their physical locations, turning the into digital, social environments that add value and personality. Client’s open and extensible, Experience as a Service (EaaS) platform has five core capabilities: Automation Support, Control Network, Content Lifecycle Management, Ad LifecycleManagement, and Experience Lifecycle Management. Challenges:  The application is exposed on the Internet, outside trusted on premise boundaries. It prevents malicious or accidental actions which can compromise security. The changes made could degrade the performance to which users are accustomed. High business complexity to design and maintain all the services Advertisement schedule setup and running associated with Hardware setup, So need more resources Achievements: Maintaining test scenarios, test reports, and use cases for every new release of firmware.  Analyzed embedded software modules to bring negative scenarios. Found at least one corner case in all modules.  Documenting scenarios and test cases in confluence. Found an issue while testing the module which doesn’t handle sound and video together. The client appreciated fordoing R&D level testing for a future-focused feature. By analyzing the functionalities, found some interesting issues which had not been fixed from so long as thosescenarios are not straight away. Ability to deliver on time. Completing tasks within the sprint with increased productivity. Ability to work on all hardware modules. Working on high-end features correlating with trending market values. Planning to implement automation for hardware. Building IoT platform for an Ad network company

Infrastructure Support – Application Servers Administration ATMECS – Content Team Challenges: Multiple versions of Application servers Application dependency Non-availability of documentation Limited alert setup for Application servers Solution Approach: Application patching on jboss , tomcat and mule for more than 5 applications Building new jboss servers for application- setting up jboss,connection & http pools, data sources, properties, MQ connections, memory sizes — for all the above configuring server groups, profiles for management purposes. Troubleshooting application issues like application unreachable, slow application and front end error responses. Troubleshooting Linux issues which effect Jboss applications like wrong permissions , mount points, file not found. Configuring monitoring on Appdynamics for new requests. Dev, QA team requests on logs, config files and deployment issues. ATMECS Contribution: Undertook server analysis for more than 50 issues/servers , which intended to reduce repeated issues by fixing them permanently. Jboss script upgradation. Inventory preparation for jboss, which was earlier undocumented Infrastructure Support – Application Servers Administration

Infrastructure Support – Cloud Administration – AWS ATMECS – Content Team Challenges: Lack of Governance in the new environment Lack of proper cloud formation templates Requirements for a hybrid cloud model for different applications Lack of proper thresholds for alerting Solution Approach: Make sure the operations run smoothly and resolve any production issues at the earliest without downtime In case of an outage, follow it up with an RCA, at the earliest Make sure that we can continuously monitor our application’s behavior and optimize alert thresholds based on historical data Make sure that every change to the infra is recorded and documented in CloudFormation Templates (Infrastructure as a code) Come up with new suggestions/recommendations as per the industry’s best practices Try to use fully managed AWS service to have minimal operations overhead ATMECS Contribution: Implemented the best practices of DevOps as well as of Agile methodology across multiple environments in AWS Implemented CI/CD pipelines involving Packer, Salt Stack, Jenkins and AWS for web app environments so that the client is confident about the changes in code and its deployment across server Created several Jenkins jobs which run across multiple AWS environments to find AWS resources (IPs | ELBs | NameTags | ENICs etc.) Created custom dashboards and configured SNS notifications for proactive alerting of any potential outages Infrastructure Support – Cloud Administration – AWS

Database Administration SQL Server ATMECS – Content Team Challenges: Limited documentation Competing priorities Limited alert setup for database functions Application dependency on lower functions Solution Approach: SQL Server Provisioning SQL Server Upgrade SQL Server Monitoring to keep alive(24*7) Disk Spaces, CPU and Memory verification Database Backups and Recovery Database refresh from higher to lower environment Database Patching Database User Management Database Indexes Debugging Application related DB issues Achieve SLAs ATMECS Contribution: Implemented a robust monitoring system to monitor SQL Services status and CPU, Memory and Disk level thresholds Placed a PowerShell script to monitor and report all the error events in production error logs Automated the process to perform DB refresh from higher to lower environments Automated the process to rebuild index offline/online on periodic basis Placed scripts to generate various types of application reports Developed jobs for Index Optimization Developed scripts to auto failover the SQL Instances Database Administration – SQL Server

Infrastructure Support – Linux Administration ATMECS – Content Team Challenges: Limited documentation Varied flavors and versions across the enterprise Space issues due to log files Elevated access for developers leading to infrastructure issues Solution Approach: Implementing the RFCs Managing the user global access through Third party AD environment Solarwinds, Linux Configuration Veritas Volume Manager Administration Server Patching and year end reboots Managing Servers through VMWare VSphere Environment ATMECS Contribution: Implementation of log rotation on multiple servers to save Disk space and which in turn helped in reducing Incident Volumes Automated Daily routine tasks to reduce 40% of the tickets Automated JBoss services Implementing ELK-stack infra in the client environment for Centralized log and Log analysis Infrastructure Support – Linux Administration

Infrastructure Support – MQ Administration ATMECS – Content Team Challenges: Limited documentation Limited alert setup for MQ issues Complicated setup for DB replication Solution Approach: Monitoring queues, channels and queue manager statistics using MQ explorer, IR360 and manually Providing the levels of access to users/applications for MQ objects such as queues, channels and Queue managers etc Creating the MQ point to point setup Creating client server architecture by creating server connection channels, client connection channels and tables ATMECS Contribution: Created alerts in all environments in case of any MQ queue manager goes down. Setup a completely new MQ queue Capture/Replication architecture on Production DB servers. Setup queue depth alerts based on thresholds Setup re-processing of dead letter queue messages by using dead letter handler Infrastructure Support – MQ Administration

Infrastructure Support – Network Administration ATMECS – Content Team Challenges: Limited or no documentation Certificate monitoring issues Complicated and manual workload for normal tasks Limited monitoring capability Solution Approach: Solarwinds, Smokeping and MRTG Monitoring NAT and Access Policy changes on Firewall Creating IPSEC tunnel on routers between Data Center and AWS VPC Creating and Managing BGP between Office and Data Center Installation of new hardware and RMA processing iOS upgrade on all Routers to the stable version based on security recommendations ATMECS Contribution: Migrated a dozen offices and three data centers firewall platforms to Palo Alto from legacy Cisco/Juniper devices, including full Palo Alto next-gen security suite implementation, all managed from Panorama Securely connected multiple AWS and Azure cloud environments into multiple data centers Implemented and configured a third party tool “OXIDISED” to take regular configuration backups from all the network devices Deployed multiple Juniper Virtual Chassis, Virtual Chassis Fabric and Qfabric data center LAN deployments Infrastructure Support – Network Administration

Infrastructure Support – Office365 Administration ATMECS – Content Team Challenges: Limited documentation Phased migration from on-prem to hybrid and then to cloud Mailbox sizes during migration to cloud 3rd party add-on integration failures during migration Solution Approach: Managing mailbox databases to accumulate storage by moving mailboxes when required Installing update rollups and security updates for CAS, HUB, UM and mailbox servers Rebooting Exchange servers to ensure equal system resource allocation for all the databases Monitoring and verifying replication health and status for the mailbox servers in DAG using exchange management shell Configured accepted domains, send connectors for the hybrid mail flow Configured directory sync in staging mode Planning mailbox migrations in phases considering various departments Performing mailbox moves to office 365 and completing the move requests as scheduled Configuring transport rules on office 365 to improve the tenant Microsoft Score Manage recipients, Groups, resource mailboxes, shared mailboxes and active sync devices Setup conditional access for Office 365 apps and Intune policies to manage ActiveSync devices Enable auto expand archive quotas, archive mailboxes and Set up retention tags and retention policies ATMECS Contribution: Suggested and implemented disabling email forwarding option from the office 365 outlook web app options to improve security Outlook settings such as safe senders list, auto download pictures for safe senders using GPO to increase usability Infrastructure Support – Office365 Administration