5

I found that after the latest leap second insertion (2016-12-31 23:59:60), our CentOS7 application that has worker threads sleeping for 1 second between jobs, started to wake the sleeping threads immediately instead of in a second. In general, all sleeps are awake 1 second ahead of expected wake time.

The simplest and working solution is to reboot the box. But that is not desirable in our case. Is there a way to fix this without rebooting?

PS. For reference, here's a simple program in C++ that reproduces the issue.

#include <boost/date_time.hpp>
#include <boost/thread.hpp>
#include <iostream>

using namespace std;


// this has to be run in a thread to be able to detect the issue
void check_thread()
{
    size_t expected_delay = 1000;
    cout << "Expected delay: " << expected_delay << " ms" << endl;
    boost::posix_time::ptime t1 = boost::posix_time::microsec_clock::universal_time();
    boost::this_thread::sleep(boost::posix_time::milliseconds(1000));
    boost::posix_time::ptime t2 = boost::posix_time::microsec_clock::universal_time();
    size_t actual_delay = (t2 - t1).total_milliseconds();
    cout << "Actual delay: " << actual_delay << " ms" << endl;
    if (abs(expected_delay - actual_delay) > 900) {
        cout << "Too big delay difference: " << (expected_delay - actual_delay) << endl;
        cout << "Possible leap second issue" << endl;
    }
    else {
        cout << "No issues found" << endl;
    }
}

int main()
{
    boost::thread_group g;
    g.create_thread(check_thread);
    g.join_all();
    return 0;
}

Building:

g++ sleep_test.cpp -Wl,-Bstatic -lboost_thread -lboost_system -lboost_date_time -Wl,-Bdynamic -rdynamic -pthread
1
  • 1
    You'll have to restart the application. In future, apply updates in a timely manner. Jan 13, 2017 at 23:28

2 Answers 2

6

Is your system time synched with ntpd or ptp? If not, update your tzdata package.

For systems not synchronized by ntpd or ptp an updated tzdata package that contains the December 31st leap second is required. The updated tzdata package was released as part of RHEA-2016-1982, and any systems using RHEL 7 that are not synchronized by ntpd or ptp should update to tzdata-2016g-2.el7, or a later version, to receive this fix.

Resolve Leap Second Issues in Red Hat Enterprise Linux

3
  • Installing a more recent tzdata-2016j-1.el7 does not fix the issue. Probably, it should be installed prior the leap second is inserted to help. Any other solutions?
    – amosk
    Jan 15, 2017 at 12:46
  • I've found another solution: set time 1 second forward, then revert it.
    – amosk
    Jan 15, 2017 at 13:28
  • 1
    @AnatolyMoskovsky If that was your solution, you can create an answer and mark it. Jan 16, 2017 at 20:10
0

In addition to what Troy said, on RHEL7 systems that had not updated tzdata by the moment the leap second is applied, and not running ntpd, an extra step is required - manually set time 1 second forward, then revert it:

date -s "+1 sec"
date -s "-1 sec"

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .