0

I work in a small product company. I am part of team of size 4 that is building a deployment pipeline for our product

My company has also hired a freelance devops consultant who helps us in managing our CI/CD platform. This guy has around 15 years of experience and is a hothead and i don't trust him.

We use jenkins CI\CD tool and have installed it on an aws ec2 instance. All the my team mates and the devops consultant has root access to the ec2 instance.

Today at 11 am suddenly jenkins UI stopped working. It was loading really slow. We restarted jenkins, increased heap size and everything we could think of, but were not able to find a solution.

We spend around 3 to 4 hours trying to debug the problem, Suddenly this guy (devops consultant) came and fixed the problem in 5 min. When i asked him what did he did, He said he removed some temp files. Being sceptical I immediately went and checked the command history

He ran following commands

8     tc qdisc del dev eth0 root
229     tc qdisc del dev lo
230     tc qdisc ls
231     tc qdisc del dev lo root
232     echo -n "CPU" "100 99 166"
233     echo -n "CPU" -n "100 190 188" -n
234     yc qdisc del dev eth0 root
235     tc qdisc del dev eth0 root
236     tc qdisc del eth0
237     ifconfig
238     tc qdisc del eth0 root
239     tc qdisc del eth0 root 1
240     tc qdisc del dev eth0 root
241     at now +38 minutes

I did a quick google search and found that tc command is used for traffic control. It is used to simulate latency in network by inducing delay or packet loss

From the above commands its looks like he deleted some rules which were causing packet loss or delay in outgoing packets.

What i understand is this guy added some rules using tc command which caused delay or packet loss because which our jenkins UI was not loading and then deleted those rules which fixed the problem.

I am a developer and have little experience in system administration and devops. Can somebody confirm this so that i can go to management and lodge a formal complaint.

2 Answers 2

1

Its not possible to tell the state of the system prior to these commands having been run and thus not possible to tell if he was removing a change he made or if the work he did was actually going to cause no changes.

Purely from the basis of what was executed it suggests that there was a traffic control in place.

Note that tc isn't just used for delaying or producing packet loss, but also for traffic re-prioritization and bandwidth allocation. It is entirely possibly what the guy was attempting to do was meant to be beneficial but screwed up somehow.

Call me cynical, but whats with the at now +38 minutes? That is clearly requesting some commands and/or a script to be executed 38 minutes later. That's not recording in the bash history of course.

It may be that there is a queueing discpipline in place again and that is what at was doing. You could try to login to this system and run tc qdisc ls to check if the default qdisc has been altered.

In any case, if this guy says he removed some temp files, I'd definitely be cynical at that - none of what hes done there is remove temp files.

I wasn't able to recognise what the echo commands were trying to manipulate. At least there is no redirection going on in the command line (the command itself suggests it should be put in a file somewhere).

I'd suggest poking around some more to see what the current qdiscs are in place.

1
  • My suspicion arises because last command ran on the bash shell was zsh. When when into the zsh shell i can only found above commands in the history. All the members in my team are developers and nobody has in depth knowledge of sysadmin or devops. I am quite sure nobody from my team have knowledge of tc. I will try to get "tc qdisc ls" by tomorrow.
    – jack
    Dec 21, 2018 at 18:16
0

Following up @Matthew Ife response, you can have a look into at spool directory and examine the files available there. On my system it is located in /var/spool/at/spool and you may see if and what is scheduled for execution in the future.

1
  • Thank you. I will definitely look at the spool directory
    – jack
    Dec 21, 2018 at 18:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .