运维笔记-服务未知原因宕机

文章内索引
[显示]

服务未知原因宕机

130测试服务器在运行几天后,发现有俩个服务挂掉了,其中一个是API,另外一个是Service-Center

首先排查API服务宕机原因

1. 首先本地目录没有异常排除JVM异常退出

启动目录中并没有hs_err_pidXXXX.log

2. 查看log4j日志,发现8月3号服务就挂掉了。

-rw-r–r– 1 root root  371487 Aug  1 23:59 HJ-Data-Api-2018-08-01.0.log
-rw-r–r– 1 root root  249120 Aug  2 23:59 HJ-Data-Api-2018-08-02.0.log
-rw-r–r– 1 root root  184540 Aug  3 16:40 HJ-Data-Api.log

3. 查看log发现,16:40 应该是接收了一个SHUTDOWN的指令,之后退出应用。

2018-08-03 16:40:32,367	Thread-9	INFO	com.netflix.loadbalancer.PollingServerListUpdater	Shutting down the Executor Pool for PollingServerListUpdater
2018-08-03 16:40:32,374	Thread-5	INFO	org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext	Closing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@3eb77ea8: startup date [Wed Aug 01 16:08:46 CST 2018]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@34340fab
2018-08-03 16:40:32,375	Thread-5	INFO	org.springframework.cloud.netflix.eureka.serviceregistry.EurekaServiceRegistry	Unregistering application HJ-Data-Api with eureka with status DOWN
2018-08-03 16:40:32,375	Thread-5	WARN	com.netflix.discovery.DiscoveryClient	Saw local status change event StatusChangeEvent [timestamp=1533285632375, current=DOWN, previous=UP]
2018-08-03 16:40:32,376	DiscoveryClient-InstanceInfoReplicator-0	INFO	com.netflix.discovery.DiscoveryClient	DiscoveryClient_HJ-DATA-API/192.168.1.130:8500: registering service...
2018-08-03 16:40:32,381	DiscoveryClient-InstanceInfoReplicator-0	INFO	com.netflix.discovery.DiscoveryClient	DiscoveryClient_HJ-DATA-API/192.168.1.130:8500 - registration status: 204
2018-08-03 16:40:32,996	Thread-5	WARN	org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext	Exception thrown from ApplicationListener handling ContextClosedEvent
java.lang.IndexOutOfBoundsException: Index: 256, Size: 256 at java.util.ArrayList.rangeCheck(ArrayList.java:653) at java.util.ArrayList.get(ArrayList.java:429) at com.hzcf.api.components.rabbitMq.RabbitMqInit.destroy(RabbitMqInit.java:127) at com.hzcf.api.ApplicationEventListener.onApplicationEvent(HjDataApiApplication.java:51) at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:393) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:347) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:991) at org.springframework.context.support.AbstractApplicationContext$2.run(AbstractApplicationContext.java:929)
2018-08-03 16:40:32,997	Thread-5	INFO	org.springframework.context.support.DefaultLifecycleProcessor	Stopping beans in phase 2147483647
2018-08-03 16:40:33,017	Thread-5	INFO	org.springframework.context.support.DefaultLifecycleProcessor	Stopping beans in phase 0
2018-08-03 16:40:33,142	Thread-5	INFO	org.springframework.context.annotation.AnnotationConfigApplicationContext	Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@4830c9b2: startup date [Wed Aug 01 16:09:14 CST 2018]; parent: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@3eb77ea8
2018-08-03 16:40:33,147	Thread-5	WARN	org.springframework.context.annotation.AnnotationConfigApplicationContext	Exception thrown from ApplicationListener handling ContextClosedEvent
com.rabbitmq.client.AlreadyClosedException: channel is already closed due to clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=OK, class-id=0, method-id=0) at com.rabbitmq.client.impl.AMQChannel.processShutdownSignal(AMQChannel.java:286) at com.rabbitmq.client.impl.ChannelN.startProcessShutdownSignal(ChannelN.java:264) at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:574) at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:508) at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:501) at com.hzcf.api.components.rabbitMq.RabbitMqInit.destroy(RabbitMqInit.java:127) at com.hzcf.api.ApplicationEventListener.onApplicationEvent(HjDataApiApplication.java:51) at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:393) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:399) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:347) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:991) at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:958) at org.springframework.cloud.context.named.NamedContextFactory.destroy(NamedContextFactory.java:76) at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:272) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:961) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.destroySingletons(FactoryBeanRegistrySupport.java:230) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:968) at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1032) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1008) at org.springframework.context.support.AbstractApplicationContext$2.run(AbstractApplicationContext.java:929)
2018-08-03 16:40:33,151	Thread-5	INFO	com.netflix.util.concurrent.ShutdownEnabledTimer	Shutdown hook removed for: NFLoadBalancer-PingTimer-192.168.1.132
2018-08-03 16:40:33,152	Thread-5	INFO	com.netflix.util.concurrent.ShutdownEnabledTimer	Exception caught (might be ok if at shutdown)
java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) at java.lang.Runtime.removeShutdownHook(Runtime.java:239) at com.netflix.util.concurrent.ShutdownEnabledTimer.cancel(ShutdownEnabledTimer.java:70) at com.netflix.loadbalancer.BaseLoadBalancer.cancelPingTask(BaseLoadBalancer.java:613) at com.netflix.loadbalancer.BaseLoadBalancer.shutdown(BaseLoadBalancer.java:864) at com.netflix.loadbalancer.DynamicServerListLoadBalancer.shutdown(DynamicServerListLoadBalancer.java:285) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:364) at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:287) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:961) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.destroySingletons(FactoryBeanRegistrySupport.java:230) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:968) at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1032) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1008) at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:958) at org.springframework.cloud.context.named.NamedContextFactory.destroy(NamedContextFactory.java:76) at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:272) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:961) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.destroySingletons(FactoryBeanRegistrySupport.java:230) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:968) at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1032) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1008) at org.springframework.context.support.AbstractApplicationContext$2.run(AbstractApplicationContext.java:929)
2018-08-03 16:40:33,168	Thread-5	INFO	org.springframework.boot.actuate.endpoint.jmx.EndpointMBeanExporter	Unregistering JMX-exposed beans on shutdown
2018-08-03 16:40:33,170	Thread-5	INFO	org.springframework.boot.actuate.endpoint.jmx.EndpointMBeanExporter	Unregistering JMX-exposed beans
2018-08-03 16:40:33,170	Thread-5	INFO	org.springframework.jmx.export.annotation.AnnotationMBeanExporter	Unregistering JMX-exposed beans on shutdown
2018-08-03 16:40:33,173	Thread-5	INFO	org.springframework.jmx.export.annotation.AnnotationMBeanExporter	Unregistering JMX-exposed beans
2018-08-03 16:40:33,173	Thread-5	INFO	com.netflix.discovery.DiscoveryClient	Shutting down DiscoveryClient ...
2018-08-03 16:40:33,174	Thread-5	INFO	com.netflix.discovery.DiscoveryClient	Unregistering ...
2018-08-03 16:40:33,201	Thread-5	INFO	com.netflix.discovery.DiscoveryClient	DiscoveryClient_HJ-DATA-API/192.168.1.130:8500 - deregister status: 200
2018-08-03 16:40:33,204	Thread-5	INFO	com.netflix.discovery.DiscoveryClient	Completed shut down of DiscoveryClient
2018-08-03 16:40:33,272	Thread-5	INFO	org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler	Shutting down ExecutorService 'poolScheduler'
2018-08-03 16:40:33,291	Thread-5	INFO	io.undertow.servlet	Destroying Spring FrameworkServlet 'dispatcherServlet'

尴尬的是我忘记了是自己调用的shutdown接口还是被系统kill的?

查看日志发现服务并未接收