运维笔记-服务未知原因宕机
文章内索引
[显示]
服务未知原因宕机
130测试服务器在运行几天后，发现有俩个服务挂掉了，其中一个是API，另外一个是Service-Center
首先排查API服务宕机原因
1. 首先本地目录没有异常排除JVM异常退出
启动目录中并没有hs_err_pidXXXX.log
2. 查看log4j日志，发现8月3号服务就挂掉了。
-rw-r–r– 1 root root 371487 Aug 1 23:59 HJ-Data-Api-2018-08-01.0.log
-rw-r–r– 1 root root 249120 Aug 2 23:59 HJ-Data-Api-2018-08-02.0.log
-rw-r–r– 1 root root 184540 Aug 3 16:40 HJ-Data-Api.log
3. 查看log发现，16：40 应该是接收了一个SHUTDOWN的指令，之后退出应用。
2018-08-03 16:40:32,367 Thread-9 INFO com.netflix.loadbalancer.PollingServerListUpdater Shutting down the Executor Pool for PollingServerListUpdater 2018-08-03 16:40:32,374 Thread-5 INFO org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext Closing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@3eb77ea8: startup date [Wed Aug 01 16:08:46 CST 2018]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@34340fab 2018-08-03 16:40:32,375 Thread-5 INFO org.springframework.cloud.netflix.eureka.serviceregistry.EurekaServiceRegistry Unregistering application HJ-Data-Api with eureka with status DOWN 2018-08-03 16:40:32,375 Thread-5 WARN com.netflix.discovery.DiscoveryClient Saw local status change event StatusChangeEvent [timestamp=1533285632375, current=DOWN, previous=UP] 2018-08-03 16:40:32,376 DiscoveryClient-InstanceInfoReplicator-0 INFO com.netflix.discovery.DiscoveryClient DiscoveryClient_HJ-DATA-API/192.168.1.130:8500: registering service... 2018-08-03 16:40:32,381 DiscoveryClient-InstanceInfoReplicator-0 INFO com.netflix.discovery.DiscoveryClient DiscoveryClient_HJ-DATA-API/192.168.1.130:8500 - registration status: 204 2018-08-03 16:40:32,996 Thread-5 WARN org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext Exception thrown from ApplicationListener handling ContextClosedEvent java.lang.IndexOutOfBoundsException: Index: 256, Size: 256 at java.util.ArrayList.rangeCheck(ArrayList.java:653) at java.util.ArrayList.get(ArrayList.java:429) at com.hzcf.api.components.rabbitMq.RabbitMqInit.destroy(RabbitMqInit.java:127) at com.hzcf.api.ApplicationEventListener.onApplicationEvent(HjDataApiApplication.java:51) at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:393) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:347) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:991) at org.springframework.context.support.AbstractApplicationContext$2.run(AbstractApplicationContext.java:929) 2018-08-03 16:40:32,997 Thread-5 INFO org.springframework.context.support.DefaultLifecycleProcessor Stopping beans in phase 2147483647 2018-08-03 16:40:33,017 Thread-5 INFO org.springframework.context.support.DefaultLifecycleProcessor Stopping beans in phase 0 2018-08-03 16:40:33,142 Thread-5 INFO org.springframework.context.annotation.AnnotationConfigApplicationContext Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@4830c9b2: startup date [Wed Aug 01 16:09:14 CST 2018]; parent: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@3eb77ea8 2018-08-03 16:40:33,147 Thread-5 WARN org.springframework.context.annotation.AnnotationConfigApplicationContext Exception thrown from ApplicationListener handling ContextClosedEvent com.rabbitmq.client.AlreadyClosedException: channel is already closed due to clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=OK, class-id=0, method-id=0) at com.rabbitmq.client.impl.AMQChannel.processShutdownSignal(AMQChannel.java:286) at com.rabbitmq.client.impl.ChannelN.startProcessShutdownSignal(ChannelN.java:264) at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:574) at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:508) at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:501) at com.hzcf.api.components.rabbitMq.RabbitMqInit.destroy(RabbitMqInit.java:127) at com.hzcf.api.ApplicationEventListener.onApplicationEvent(HjDataApiApplication.java:51) at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:393) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:399) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:347) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:991) at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:958) at org.springframework.cloud.context.named.NamedContextFactory.destroy(NamedContextFactory.java:76) at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:272) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:961) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.destroySingletons(FactoryBeanRegistrySupport.java:230) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:968) at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1032) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1008) at org.springframework.context.support.AbstractApplicationContext$2.run(AbstractApplicationContext.java:929) 2018-08-03 16:40:33,151 Thread-5 INFO com.netflix.util.concurrent.ShutdownEnabledTimer Shutdown hook removed for: NFLoadBalancer-PingTimer-192.168.1.132 2018-08-03 16:40:33,152 Thread-5 INFO com.netflix.util.concurrent.ShutdownEnabledTimer Exception caught (might be ok if at shutdown) java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) at java.lang.Runtime.removeShutdownHook(Runtime.java:239) at com.netflix.util.concurrent.ShutdownEnabledTimer.cancel(ShutdownEnabledTimer.java:70) at com.netflix.loadbalancer.BaseLoadBalancer.cancelPingTask(BaseLoadBalancer.java:613) at com.netflix.loadbalancer.BaseLoadBalancer.shutdown(BaseLoadBalancer.java:864) at com.netflix.loadbalancer.DynamicServerListLoadBalancer.shutdown(DynamicServerListLoadBalancer.java:285) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:364) at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:287) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:961) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.destroySingletons(FactoryBeanRegistrySupport.java:230) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:968) at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1032) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1008) at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:958) at org.springframework.cloud.context.named.NamedContextFactory.destroy(NamedContextFactory.java:76) at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:272) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:961) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.destroySingletons(FactoryBeanRegistrySupport.java:230) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:968) at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1032) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1008) at org.springframework.context.support.AbstractApplicationContext$2.run(AbstractApplicationContext.java:929) 2018-08-03 16:40:33,168 Thread-5 INFO org.springframework.boot.actuate.endpoint.jmx.EndpointMBeanExporter Unregistering JMX-exposed beans on shutdown 2018-08-03 16:40:33,170 Thread-5 INFO org.springframework.boot.actuate.endpoint.jmx.EndpointMBeanExporter Unregistering JMX-exposed beans 2018-08-03 16:40:33,170 Thread-5 INFO org.springframework.jmx.export.annotation.AnnotationMBeanExporter Unregistering JMX-exposed beans on shutdown 2018-08-03 16:40:33,173 Thread-5 INFO org.springframework.jmx.export.annotation.AnnotationMBeanExporter Unregistering JMX-exposed beans 2018-08-03 16:40:33,173 Thread-5 INFO com.netflix.discovery.DiscoveryClient Shutting down DiscoveryClient ... 2018-08-03 16:40:33,174 Thread-5 INFO com.netflix.discovery.DiscoveryClient Unregistering ... 2018-08-03 16:40:33,201 Thread-5 INFO com.netflix.discovery.DiscoveryClient DiscoveryClient_HJ-DATA-API/192.168.1.130:8500 - deregister status: 200 2018-08-03 16:40:33,204 Thread-5 INFO com.netflix.discovery.DiscoveryClient Completed shut down of DiscoveryClient 2018-08-03 16:40:33,272 Thread-5 INFO org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler Shutting down ExecutorService 'poolScheduler' 2018-08-03 16:40:33,291 Thread-5 INFO io.undertow.servlet Destroying Spring FrameworkServlet 'dispatcherServlet'
尴尬的是我忘记了是自己调用的shutdown接口还是被系统kill的？
查看日志发现服务并未接收