Nov 3, 2020 - 시간 동기화 문제

* * * * * root /usr/sbin/ntpdate -b time.bora.net
* * * * * rdate -s time.bora.net

rdate - 원격타임서버로 부터 날짜시간정보 가져오기
rdate는 지정한 원격지의 타임서버로 부터 날짜시간정보를 받아와 보여주거나 날짜시간설정을 하는 명령어입니다.

NTP는 Network Time Protocol 약자로서 rdate와 비슷한 기능을 제공합니다.
ntpdate는 rdate를 이용한 방법보다 시간을 0.01초 이하의 오차로 맞출 수 있습니다.

각 서비스에서 테스트를 진행하는데,
시간 오차가 발생하여 확인해본 결과,
특정 서버에서 rdate 가 설치가 안되어있는 이슈를 발견하여 이력을 남깁니다.

Nov 3, 2020 - 2가지의 dump 작업

jmap 으로 Heap dump 처리

jmap -dump:format=b,file=./dump_file 10372
{파일명} {pid} 으로 jmap 생성.

[WAS-05:root]/root>#ps -ef|grep java
root     10372     1 99 Nov02 ?        6-10:01:40 /usr/local/java/bin/java -Djava.util.rg.apache.juli.ClassLoaderLogManager -verbosegc -server -Xms10g -Xmx10g -XX:+UseG1GC -D-Duser.timezone=GMT+09:00 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/tg.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=00al/tomcat9/lib/scouter.agent.jar -Dcom.sun.management.jmxremote -Dcom.sun.management.jm=false -Dcom.sun.management.jmxremote.authenticate=false -Dignore.endorsed.dirs= -classase=/usr/local/tomcat9 -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/to
root     27084 26857  0 10:52 pts/1    00:00:00 grep java

[WAS-05:root]/usr/local/tomcat9/logs>#cd /home/users/bymin/
[WAS-05:root]/home/users/bymin>#jmap -dump:format=b,file=./dump_file 10372
Heap dump file created

Thread dump 처리

[WAS-05:root]/home/users/bymin>#jcmd 10372 Thread.print > ./thread_file

jcmd {pid} Thread.print > {파일명}

Thread 분석 무료 사이트 를 이용하여 분석을 완료 합니다.

Oct 27, 2020 - E11000 duplicate key

2020-10-26T22:15:29.033+0900 I SHARDING [chunkInserter] Batch insertion failed  :: caused by :: DuplicateKey: E11000 duplicate key error collection: user_mapping.auth_id_info index: _id_ dup key: { : ObjectId('5d5e75b2b26437e191d4040b') }
2020-10-26T22:15:55.692+0900 I SHARDING [chunkInserter] Batch insertion failed  :: caused by :: DuplicateKey: E11000 duplicate key error collection: user_mapping.auth_id_info index: _id_ dup key: { : ObjectId('5d5e75b2b26437e191d4040b') }
2020-10-26T22:16:22.884+0900 I SHARDING [chunkInserter] Batch insertion failed  :: caused by :: DuplicateKey: E11000 duplicate key error collection: user_mapping.auth_id_info index: _id_ dup key: { : ObjectId('5d5e75b2b26437e191d4040b') }
2020-10-26T22:16:50.175+0900 I SHARDING [chunkInserter] Batch insertion failed  :: caused by :: DuplicateKey: E11000 duplicate key error collection: user_mapping.auth_id_info index: _id_ dup key: { : ObjectId('5d5e75b2b26437e191d4040b') }
2020-10-26T22:17:17.405+0900 I SHARDING [chunkInserter] Batch insertion failed  :: caused by :: DuplicateKey: E11000 duplicate key error collection: user_mapping.auth_id_info index: _id_ dup key: { : ObjectId('5d5e75b2b26437e191d4040b') }
2020-10-26T22:17:44.653+0900 I SHARDING [chunkInserter] Batch insertion failed  :: caused by :: DuplicateKey: E11000 duplicate key error collection: user_mapping.auth_id_info index: _id_ dup key: { : ObjectId('5d5e75b2b26437e191d4040b') }
2020-10-26T22:18:10.891+0900 I SHARDING [chunkInserter] Batch insertion failed  :: caused by :: DuplicateKey: E11000 duplicate key error collection: user_mapping.auth_id_info index: _id_ dup key: { : ObjectId('5d5e75b2b26437e191d4040b') }
2020-10-26T22:18:37.264+0900 I SHARDING [chunkInserter] Batch insertion failed  :: caused by :: DuplicateKey: E11000 duplicate key error collection: user_mapping.auth_id_info index: _id_ dup key: { : ObjectId('5d5e75b2b26437e191d4040b') }
2020-10-26T22:19:04.515+0900 I SHARDING [chunkInserter] Batch insertion failed  :: caused by :: DuplicateKey: E11000 duplicate key error collection: user_mapping.auth_id_info index: _id_ dup key: { : ObjectId('5d5e75b2b26437e191d4040b') }
2020-10-26T22:19:31.047+0900 I SHARDING [chunkInserter] Batch insertion failed  :: caused by :: DuplicateKey: E11000 duplicate key error collection: user_mapping.auth_id_info index: _id_ dup key: { : ObjectId('5d5e75b2b26437e191d4040b') }

db.getCollection(‘auth_id_info’).find({_id:ObjectId(“5d5e75b2b26437e191d4040b”)}) 로 검색시, 두개의 값이 나오는 것을 확인할 수 있었으며,

해당 키를 삭제하면 해결됩니다.

다만, 키가 동일하므로 두개의 값이 다 삭제될 수 있으므로, 해당 데이터를 백업해야할 수 있습니다!