Team Atmecs

Infrastructure Support – Application Servers Administration ATMECS – Content Team Challenges: Multiple versions of Application servers Application dependency Non-availability of documentation Limited alert setup for Application servers Solution Approach: Application patching on jboss , tomcat and mule for more than 5 applications Building new jboss servers for application- setting up jboss,connection & http pools, data sources, properties, MQ connections, memory sizes — for all the above configuring server groups, profiles for management purposes. Troubleshooting application issues like application unreachable, slow application and front end error responses. Troubleshooting Linux issues which effect Jboss applications like wrong permissions , mount points, file not found. Configuring monitoring on Appdynamics for new requests. Dev, QA team requests on logs, config files and deployment issues. ATMECS Contribution: Undertook server analysis for more than 50 issues/servers , which intended to reduce repeated issues by fixing them permanently. Jboss script upgradation. Inventory preparation for jboss, which was earlier undocumented Infrastructure Support – Application Servers Administration

Infrastructure Support – Cloud Administration – AWS ATMECS – Content Team Challenges: Lack of Governance in the new environment Lack of proper cloud formation templates Requirements for a hybrid cloud model for different applications Lack of proper thresholds for alerting Solution Approach: Make sure the operations run smoothly and resolve any production issues at the earliest without downtime In case of an outage, follow it up with an RCA, at the earliest Make sure that we can continuously monitor our application’s behavior and optimize alert thresholds based on historical data Make sure that every change to the infra is recorded and documented in CloudFormation Templates (Infrastructure as a code) Come up with new suggestions/recommendations as per the industry’s best practices Try to use fully managed AWS service to have minimal operations overhead ATMECS Contribution: Implemented the best practices of DevOps as well as of Agile methodology across multiple environments in AWS Implemented CI/CD pipelines involving Packer, Salt Stack, Jenkins and AWS for web app environments so that the client is confident about the changes in code and its deployment across server Created several Jenkins jobs which run across multiple AWS environments to find AWS resources (IPs | ELBs | NameTags | ENICs etc.) Created custom dashboards and configured SNS notifications for proactive alerting of any potential outages Infrastructure Support – Cloud Administration – AWS

Database Administration SQL Server ATMECS – Content Team Challenges: Limited documentation Competing priorities Limited alert setup for database functions Application dependency on lower functions Solution Approach: SQL Server Provisioning SQL Server Upgrade SQL Server Monitoring to keep alive(24*7) Disk Spaces, CPU and Memory verification Database Backups and Recovery Database refresh from higher to lower environment Database Patching Database User Management Database Indexes Debugging Application related DB issues Achieve SLAs ATMECS Contribution: Implemented a robust monitoring system to monitor SQL Services status and CPU, Memory and Disk level thresholds Placed a PowerShell script to monitor and report all the error events in production error logs Automated the process to perform DB refresh from higher to lower environments Automated the process to rebuild index offline/online on periodic basis Placed scripts to generate various types of application reports Developed jobs for Index Optimization Developed scripts to auto failover the SQL Instances Database Administration – SQL Server

Infrastructure Support – Linux Administration ATMECS – Content Team Challenges: Limited documentation Varied flavors and versions across the enterprise Space issues due to log files Elevated access for developers leading to infrastructure issues Solution Approach: Implementing the RFCs Managing the user global access through Third party AD environment Solarwinds, Linux Configuration Veritas Volume Manager Administration Server Patching and year end reboots Managing Servers through VMWare VSphere Environment ATMECS Contribution: Implementation of log rotation on multiple servers to save Disk space and which in turn helped in reducing Incident Volumes Automated Daily routine tasks to reduce 40% of the tickets Automated JBoss services Implementing ELK-stack infra in the client environment for Centralized log and Log analysis Infrastructure Support – Linux Administration

Infrastructure Support – MQ Administration ATMECS – Content Team Challenges: Limited documentation Limited alert setup for MQ issues Complicated setup for DB replication Solution Approach: Monitoring queues, channels and queue manager statistics using MQ explorer, IR360 and manually Providing the levels of access to users/applications for MQ objects such as queues, channels and Queue managers etc Creating the MQ point to point setup Creating client server architecture by creating server connection channels, client connection channels and tables ATMECS Contribution: Created alerts in all environments in case of any MQ queue manager goes down. Setup a completely new MQ queue Capture/Replication architecture on Production DB servers. Setup queue depth alerts based on thresholds Setup re-processing of dead letter queue messages by using dead letter handler Infrastructure Support – MQ Administration

Infrastructure Support – Network Administration ATMECS – Content Team Challenges: Limited or no documentation Certificate monitoring issues Complicated and manual workload for normal tasks Limited monitoring capability Solution Approach: Solarwinds, Smokeping and MRTG Monitoring NAT and Access Policy changes on Firewall Creating IPSEC tunnel on routers between Data Center and AWS VPC Creating and Managing BGP between Office and Data Center Installation of new hardware and RMA processing iOS upgrade on all Routers to the stable version based on security recommendations ATMECS Contribution: Migrated a dozen offices and three data centers firewall platforms to Palo Alto from legacy Cisco/Juniper devices, including full Palo Alto next-gen security suite implementation, all managed from Panorama Securely connected multiple AWS and Azure cloud environments into multiple data centers Implemented and configured a third party tool “OXIDISED” to take regular configuration backups from all the network devices Deployed multiple Juniper Virtual Chassis, Virtual Chassis Fabric and Qfabric data center LAN deployments Infrastructure Support – Network Administration

Infrastructure Support – Office365 Administration ATMECS – Content Team Challenges: Limited documentation Phased migration from on-prem to hybrid and then to cloud Mailbox sizes during migration to cloud 3rd party add-on integration failures during migration Solution Approach: Managing mailbox databases to accumulate storage by moving mailboxes when required Installing update rollups and security updates for CAS, HUB, UM and mailbox servers Rebooting Exchange servers to ensure equal system resource allocation for all the databases Monitoring and verifying replication health and status for the mailbox servers in DAG using exchange management shell Configured accepted domains, send connectors for the hybrid mail flow Configured directory sync in staging mode Planning mailbox migrations in phases considering various departments Performing mailbox moves to office 365 and completing the move requests as scheduled Configuring transport rules on office 365 to improve the tenant Microsoft Score Manage recipients, Groups, resource mailboxes, shared mailboxes and active sync devices Setup conditional access for Office 365 apps and Intune policies to manage ActiveSync devices Enable auto expand archive quotas, archive mailboxes and Set up retention tags and retention policies ATMECS Contribution: Suggested and implemented disabling email forwarding option from the office 365 outlook web app options to improve security Outlook settings such as safe senders list, auto download pictures for safe senders using GPO to increase usability Infrastructure Support – Office365 Administration

Infrastructure Support – Sharepoint Administration ATMECS – Content Team Challenges: Limited documentation Multiple versions across applications including out-of-support ones Lack of high availability Lots of custom workflows and customization requirements Solution Approach: Day to day Activities, User Management Scheduled backups Moving Applications to different environments(DEV, QA, UAT & PROD) Implement High availability (Database level) Monitoring Server health Verifying Memory & CPU utilization Configured Remote BLOB Storage (RBS) Identifying Server patches and testing on DEV -Environment Deploying WSPs in all environments(DEV, QA, UAT & PROD) Configuring Search Application and Crawl configurations Identifying Application/Server/search/Crawl issues ATMECS Contribution: Wrote the reusable PowerShell scripts Consultancy on developing multiple custom work-flows Resolved long-standing Enterprise search issues Provided recommendations for Patch installation Completed feasibility study based on moving existing applications to AWS and Azure and provided recommendations Infrastructure Support – Sharepoint Administration

Infrastructure Support – Storage Administration ATMECS – Content Team Challenges: Limited documentation Lots of redundant backups being performed Setting up of appliance on AWS Issues with deduplication ratios due to DB compressed backups NetApp capacity issues due to skewed loads Shared folder issues due to user error Solution Approach: Storage team advised the Customer about the Volume movement feature and implementing to the environment after Customer approval. We will start planning to migrate volumes where more high I/O operations and more Storage Utilization. Storage Team will start migrating the volumes to other nodes after validating space and CPU utilization. During Production volume migration, Volume move will initially migrate the source data to the destination storage and pass it under cutover phase. We start communicating with Customer for the data cutover during Scheduled downtime to the dedicated Storage. ATMECS Contribution: SQL Server database of a particular application was protected by livevault backup solution and it costs 1500$ monthly per TB. To reduce the cost effect, we implemented NetBackup backup protection for these servers. Since these servers are hosted on AWS cloud, we built NetBackup Master server with S3 bucket storage unit on AWS to achieve better performance with minimum cost effect. In future, we are looking to add more clients under this AWS master server, which application server are in AWS platform for better server management. Infrastructure Support Storage Administration

Infrastructure Support – Telecom Administration ATMECS – Content Team Challenges: Limited or no documentation Migration from Avaya to Cisco Integration of the additional departments to the environment Limited monitoring capability Solution Approach: Migration from Avaya telephony system to Cisco MACD (Move, add, change and delete) Troubleshooting of IP phone registration issues, softphone issues & call routing Issues Monitoring call signaling protocols on Voice Gateways like MGCP, SIP Troubleshooting media resource issues in transcoder, MOH, conference bridges, IPT voice quality issues between and within locations Upgrading, adding licenses and troubleshooting all aspects of CUCM along with intimate usage of the tools CUCM offers (RTMT, DNA)Manual CUCM Backup if required. ATMECS Contribution: Integrated the alternate monitoring tool (RTMT) to keep track of VoIP activity including all the UC application Infrastructure Support Telecom Administration