SonarQube MySQL read time out -
my system consists of:
sonarqube 5.1, mysql 5.6.19, sonar-qube runner 2.4
i have project 20 submodules , 2,7 million lines of code.
when try execute analysis randomly fails following error:
error: caused by: unable request: /batch/project?key=xxx error: caused by: read timed out
log-file:
2015.06.19 11:25:20 trace web[sql] time=2ms | sql=select * `users` (login='admin' , active=1) limit 1 2015.06.19 11:25:20 trace web[sql] time=1ms | sql=select * `groups` inner join `groups_users` on `groups`.id = `groups_users`.group_id (`groups_users`.user_id = 1 ) 2015.06.19 11:25:20 debug web[http] /api/server | time=42ms 2015.06.19 11:25:22 trace web[sql] time=1ms | sql=select * `users` (login='admin' , active=1) limit 1 2015.06.19 11:25:22 trace web[sql] time=1ms | sql=select * `groups` inner join `groups_users` on `groups`.id = `groups_users`.group_id (`groups_users`.user_id = 1 ) 2015.06.19 11:25:22 trace web[sql] time=3ms | sql=select gr.role group_roles gr inner join groups_users gu on gu.group_id=gr.group_id inner join users u on u.id=gu.user_id u.login=? , gr.resource_id null union select gr.role group_roles gr gr.group_id null , gr.resource_id null union select ur.role user_roles ur inner join users u on u.id=ur.user_id u.login=? , ur.resource_id null 2015.06.19 11:25:22 trace web[sql] time=1ms | sql=select p.id, p.uuid uuid, p.project_uuid projectuuid, p.module_uuid moduleuuid, p.module_uuid_path moduleuuidpath, p.kee kee, p.deprecated_kee deprecatedkey, p.name name, p.long_name longname, p.qualifier qualifier, p.scope scope, p.language language, p.root_id parentprojectid, p.path path, p.enabled enabled, p.copy_resource_id copyresourceid, p.authorization_updated_at authorizationupdatedat, p.created_at createdat projects p p.kee=? 2015.06.19 11:25:29 trace web[sql] time=1ms | sql=select ar.id analysis_reports ar ar.report_status=? , not exists( select 1 analysis_reports ar2 ar.project_key = ar2.project_key , ar2.report_status=? ) order ar.created_at asc, ar.id asc 2015.06.19 11:25:29 trace web[sql] time=7105ms | sql=select p.id, p.uuid uuid, p.project_uuid projectuuid, p.module_uuid moduleuuid, p.module_uuid_path moduleuuidpath, p.kee kee, p.deprecated_kee deprecatedkey, p.name name, p.long_name longname, p.qualifier qualifier, p.scope scope, p.language language, p.root_id parentprojectid, p.path path, p.enabled enabled, p.copy_resource_id copyresourceid, p.authorization_updated_at authorizationupdatedat, p.created_at createdat projects p inner join projects module on module.project_uuid = p.project_uuid , module.uuid = ? , module.scope='prj' , module.enabled = true p.enabled = true , p.scope = ? , p.module_uuid_path concat(module.module_uuid_path, '%') 2015.06.19 11:25:36 trace web[sql] time=6092ms | sql=select prop.id id, prop.prop_key "key", prop.text_value value, prop.resource_id resourceid, prop.user_id userid properties prop inner join (select p.id projects p inner join projects module on module.project_uuid = p.project_uuid , module.uuid = ? , module.scope='prj' , module.enabled = true p.enabled = true , p.scope = ? , p.module_uuid_path concat(module.module_uuid_path, '%') ) modules on modules.id=prop.resource_id prop.user_id null 2015.06.19 11:25:39 trace web[sql] time=1ms | sql=select ar.id analysis_reports ar ar.report_status=? , not exists( select 1 analysis_reports ar2 ar.project_key = ar2.project_key , ar2.report_status=? ) order ar.created_at asc, ar.id asc 2015.06.19 11:25:48 warn es[o.e.c.r.a.decider] [sonar-1434705179964] high disk watermark [10%] exceeded on [ksolp6gbtqmey19kqxz4qa][sonar-1434705179964] free: 5.5gb[9.3%], shards relocated away node 2015.06.19 11:25:49 trace web[sql] time=0ms | sql=select ar.id analysis_reports ar ar.report_status=? , not exists( select 1 analysis_reports ar2 ar.project_key = ar2.project_key , ar2.report_status=? ) order ar.created_at asc, ar.id asc 2015.06.19 11:25:49 trace web[sql] time=13656ms | sql=select p.uuid, p.path, p.module_uuid moduleuuid, fs.src_hash srchash projects p inner join file_sources fs on fs.file_uuid=p.uuid p.project_uuid=? , p.enabled=true , p.scope='fil' 2015.06.19 11:25:50 trace web[sql] time=2ms | sql=select p.id id, p.kee kee, p.name name, p.language language, p.parent_kee parentkee, p.created_at createdat, p.updated_at updatedat, p.rules_updated_at rulesupdatedat rules_profiles p, projects project inner join properties prop on prop.resource_id=project.id p.language=? , prop.prop_key ? , prop.text_value p.name , project.kee=? 2015.06.19 11:25:50 trace web[sql] time=2ms | sql=select p.id id, p.kee kee, p.name name, p.language language, p.parent_kee parentkee, p.created_at createdat, p.updated_at updatedat, p.rules_updated_at rulesupdatedat rules_profiles p inner join properties prop on prop.prop_key=? , prop.resource_id null , prop.text_value p.name , p.language=? 2015.06.19 11:25:50 trace web[es] es search request '{"from":0,"size":100,"query":{"filtered":{"query":{"match_all":{}},"filter":{"bool":{"must":[{"bool":{"must_not":{"term":{"status":"removed"}}}},{"terms":{"lang":["java"]}}]}}}},"_source":{"includes":["templatekey","internalkey","name","key"],"excludes":[]},"sort":[{"updatedat":{"order":"desc"}},{"key.sort":{"order":"asc"}}]}' on indices '[rules]' on types '[rule]' | time=42ms 2015.06.19 11:25:50 trace web[es] es search scroll request scroll id 'org.elasticsearch.search.scroll@4965453a' | time=107ms 2015.06.19 11:25:50 trace web[es] es search scroll request scroll id 'org.elasticsearch.search.scroll@17a4087b' | time=55ms 2015.06.19 11:25:50 trace web[es] es search scroll request scroll id 'org.elasticsearch.search.scroll@1228cf5d' | time=100ms 2015.06.19 11:25:50 trace web[es] es search scroll request scroll id 'org.elasticsearch.search.scroll@3545ebfb' | time=61ms 2015.06.19 11:25:50 trace web[es] es search scroll request scroll id 'org.elasticsearch.search.scroll@187c710' | time=47ms 2015.06.19 11:25:50 trace web[es] es search scroll request scroll id 'org.elasticsearch.search.scroll@791dec38' | time=46ms 2015.06.19 11:25:50 trace web[es] es search scroll request scroll id 'org.elasticsearch.search.scroll@1b7f8ea7' | time=24ms 2015.06.19 11:25:50 trace web[es] es search scroll request scroll id 'org.elasticsearch.search.scroll@536e840b' | time=23ms 2015.06.19 11:25:50 trace web[es] es search scroll request scroll id 'org.elasticsearch.search.scroll@7c9f18f7' | time=23ms 2015.06.19 11:25:50 trace web[es] es search scroll request scroll id 'org.elasticsearch.search.scroll@3120400c' | time=22ms 2015.06.19 11:25:50 trace web[es] es search scroll request scroll id 'org.elasticsearch.search.scroll@3c4ced06' | time=22ms 2015.06.19 11:25:50 trace web[es] es search scroll request scroll id 'org.elasticsearch.search.scroll@5e5d43d7' | time=24ms 2015.06.19 11:25:50 trace web[es] es search scroll request scroll id 'org.elasticsearch.search.scroll@3d1108a7' | time=23ms 2015.06.19 11:25:50 trace web[es] es search scroll request scroll id 'org.elasticsearch.search.scroll@294f2b43' | time=14ms 2015.06.19 11:25:50 trace web[es] es search scroll request scroll id 'org.elasticsearch.search.scroll@48dbf596' | time=4ms 2015.06.19 11:25:50 trace web[es] es search request '{"size":100,"query":{"filtered":{"query":{"match_all":{}},"filter":{"bool":{"must":{"term":{"profile":"java-r7-quality-profile-66231"}},"must_not":{"has_parent":{"filter":{"term":{"status":"removed"}},"parent_type":"rule"}}}}}}}' on indices '[rules]' on types '[activerule]' | time=79ms 2015.06.19 11:25:50 trace web[es] es search scroll request scroll id 'org.elasticsearch.search.scroll@493ff1bd' | time=8ms 2015.06.19 11:25:50 trace web[es] es search scroll request scroll id 'org.elasticsearch.search.scroll@5e7eb49f' | time=2ms 2015.06.19 11:25:50 trace web[es] es search request '{"from":0,"size":10,"query":{"filtered":{"query":{"match_all":{}},"filter":{"bool":{"must":[{"bool":{"must_not":{"term":{"status":"removed"}}}},{"terms":{"repo":["manual"]}}]}}}},"_source":{"includes":["name","key"],"excludes":[]},"sort":[{"updatedat":{"order":"desc"}},{"key.sort":{"order":"asc"}}]}' on indices '[rules]' on types '[rule]' | time=4ms 2015.06.19 11:25:50 trace web[es] es search scroll request scroll id 'org.elasticsearch.search.scroll@50846330' | time=2ms 2015.06.19 11:25:50 trace web[es] es search scroll request scroll id 'org.elasticsearch.search.scroll@238c0d9c' | time=2ms 2015.06.19 11:25:50 debug web[http] /batch/project?key=t3000-build-latest&preview=false | **time=28226ms** 2015.06.19 11:25:57 trace web[sql] time=1ms | sql=select id, data notifications order id asc limit ? 2015.06.19 11:25:59 trace web[sql] time=1ms | sql=select ar.id analysis_reports ar ar.report_status=? , not exists( select 1 analysis_reports ar2 ar.project_key = ar2.project_key , ar2.report_status=? ) order ar.created_at asc, ar.id asc 2015.06.19 11:26:09 trace web[sql] time=1ms | sql=select ar.id analysis_reports ar ar.report_status=? , not exists( select 1 analysis_reports ar2 ar.project_key = ar2.project_key , ar2.report_status=? ) order ar.created_at asc, ar.id asc 2015.06.19 11:26:18 warn es[o.e.c.r.a.decider] [sonar-1434705179964] high disk watermark [10%] exceeded on [ksolp6gbtqmey19kqxz4qa][sonar-1434705179964] free: 5.5gb[9.3%], shards relocated away node 2015.06.19 11:26:18 info es[o.e.c.r.a.decider] [sonar-1434705179964] high disk watermark exceeded on 1 or more nodes, rerouting shards 2015.06.19 11:26:19 trace web[sql] time=4ms | sql=select ar.id analysis_reports ar ar.report_status=? , not exists( select 1 analysis_reports ar2 ar.project_key = ar2.project_key , ar2.report_status=? ) order ar.created_at asc, ar.id asc i know fails, because read time > 20 ms. how can increase read time? or else can do, error not come anymore?
this known limitation can indeed encounter under conjunction of circumstances (project lots of modules and/or files, heavy load on sq server, ...).
there open ticket on limitation, , we'll working on this: sonar-6604.
Comments
Post a Comment