trap invalid opcode ip:fea6f9

[Smbc1]

Strange error crushes all node processes on AWS EC2 instance without change to catch it. And, of course, it forces instance to restart. Previously (in node 0.6) I have seen error "invalid opcode 0000", but this one seems different (node 5.0.0).
Full error text is

trap invalid opcode ip:fea6f9 sp:7ffcdc661708 error:0 in node[400000+1306000]

[bnoordhuis]

Does it happen right away at startup? Does your application use native add-ons directly or indirectly? If find node_modules -name \*.node finds anything, the answer is yes.

[santigimeno]

Yesterday, I was getting similar errors intermittently when running the tests from master

trap invalid opcode ip:e2c519 sp:7fff385218d8 error:0 in node[400000+e18000]

It's a Jessie 64bit box.

[Smbc1]

Yes, I am using some native add-ons, but error appears not on startup, but in work under few hundreds connections.

[Smbc1]

In fact, it appears a few times in day. I am using weak, bufferutil, kerberos, utf-8-validate, segfault-handler and some other native modules. Maybe it's bad idea to use segfault-handler in release, but I am doing it because of #3715

[bnoordhuis]

Can you turn on core dumps with ulimit -c unlimited? Once you have a core dump, try loading it in gdb and checking the stack trace:

$ gdb node core
> thread apply all backtrace full
# post what's printed

[Smbc1]

The problem is that error happens only in release, which is EC2 autoscaled instance, so I have no time to call "thread apply all backtrace full" before instance will be terminated.

[Smbc1]

But I'll try it on one instance, hope it'll be useful.

[santigimeno]

I don't know if it's the same issue or an issue at all, but running test-domain-with-abort-on-uncaught-exception.js causes a trap invalid opcode ip:e2c519 kernel trace in my machine. The process does not crash though. Specifically running:

testDomainExceptionHandling('--abort_on_uncaught_exception', {
    throwInDomainErrHandler: true,
    useTryCatch: false
});

[bnoordhuis]

@santigimeno That's expected. The test starts a child process that is expected to abort. The 'invalid opcode' is the UD2 instruction that the child process uses to terminate itself.

[santigimeno]

@bnoordhuis I understand. Thanks a lot for the info. I didn't know about it

[Smbc1]

@bnoordhuis, I got it, it's pretty short...

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `next engine worker                                               '.
Program terminated with signal 4, Illegal instruction.
#0  0x0000000000fea6f9 in v8::base::OS::Abort() ()

[bnoordhuis]

Did you run thread apply all backtrace full in gdb?

[Smbc1]

Sorry, @bnoordhuis, my fault. This one:

warning: core file may not match specified executable file.
[New LWP 4375]
[New LWP 4376]
[New LWP 4380]
[New LWP 4379]
[New LWP 4382]
[New LWP 4378]
[New LWP 4383]
[New LWP 4381]
[New LWP 4377]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `next engine worker                                               '.
Program terminated with signal 4, Illegal instruction.
#0  0x0000000000fea6f9 in v8::base::OS::Abort() ()
(gdb) thread apply all backtrace full

Thread 9 (Thread 0x7f585abf3700 (LWP 4377)):
#0  0x00007f585b78d490 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1  0x0000000000fe9518 in v8::base::Semaphore::Wait() ()
No symbol table info available.
#2  0x0000000000e878a9 in v8::platform::TaskQueue::GetNext() ()
No symbol table info available.
#3  0x0000000000e879fc in v8::platform::WorkerThread::Run() ()
No symbol table info available.
#4  0x0000000000fea4d0 in v8::base::ThreadEntry(void*) ()
No symbol table info available.
#5  0x00007f585b786b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#6  0x00007f585b4d095d in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#7  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 8 (Thread 0x7f58520dc700 (LWP 4381)):
#0  0x00007f585b78b344 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1  0x0000000000fdf8e9 in uv_cond_wait (cond=cond@entry=0x192cfc0, mutex=mutex@entry=0x192cf80) at ../deps/uv/src/unix/thread.c:380
No locals.
#2  0x0000000000fd1148 in worker (arg=arg@entry=0x0) at ../deps/uv/src/threadpool.c:75
        w = <optimized out>
        q = <optimized out>
#3  0x0000000000fdf449 in uv__thread_start (arg=<optimized out>) at ../deps/uv/src/unix/thread.c:49
        ctx_p = <optimized out>
        ctx = {entry = 0xfd1100 <worker>, arg = 0x0}
#4  0x00007f585b786b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#5  0x00007f585b4d095d in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#6  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 7 (Thread 0x7f58510da700 (LWP 4383)):
#0  0x00007f585b78b344 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0

[bnoordhuis]

Was the output clipped? 9 threads were started but the backtrace only shows 3.