SWA (Stochastic Weight Averaging)

  • 신경망 모델 다수를 앙상블링 하는 방법은 대회등에서 꾸준히 정확도를 높이기 위해 사용되어왔던 방법
  • 다수의 여러개의 모델을 만들기 위해서는 하나를 만들때보다 N배의 컴퓨팅 자원 필요. 이를 효율적으로 학습하고자 한번의 학습으로만 다양한 모델을 반영하도록 유도
    • Snapshot Ensembling – SGD가 local optima (국소최적화) 지점을 찾기위해 계속 공간을 돌아다니도록 하고, 이를 반복하여 다양한 가중치들을 찾고, 이를 평균냄
    • Fast Geometric Ensembling (FGE) – 훈련율 (learning rate)를 consine대신 linear하게 설정하여 조금더 빠르게 학습이 될수 있도록 함. 여러 골짜기(local optima)들이 중간중간 박혀있는 것이 아니라, 보통은 계곡이 계속되기 때문
  • SWA에서는 굳이 여러 모델들의 파라미터를 가지고 있지않고, 프로그래밍에서 값을 업데이트 하는 것처럼 weight를 업데이트 한 다음, 최종적으로 돌때는 이미 업데이트가 되어있는 방식
여러개의 국소값들을 찾아 이를 합치는 방식 (Snapshot Ensembling)
실질적인 국소 최적화 지점 (Fast Geometric Ensembling)
Stochastic Weight Averagin 에서 한번씩 값을 업데이트 하는 방법

15 thoughts on “SWA (Stochastic Weight Averaging)

  1. Do you mind if I quote a few of g
    your posts as long as I provide credit and sources back
    to your weblog? My website is in the very same niche as yours and my users would certainly benefit from a lot of the information you present here.
    Please let me know if this ok with you. Appreciate it!

  2. Hi there! I’m at work surfing around your blog from my new iphone!
    Just wanted to say I love reading through your blog and look forward to all
    your posts! Keep up the excellent work!

    Here is my web site g see

  3. What’s up, everything is going perfectly here and ofcourse every one is sharing facts,
    that’s in fact good, keep up writing.

    my web-site: cbd oil (tinyurl.com)

  4. hey there and thank you for your information – I’ve certainly picked up something new from right here.
    I did however expertise a few technical points using this website, since
    I experienced to reload the site lots of times previous to I could get it to load properly.
    I had been wondering if your web hosting is OK? Not that I’m complaining, but slow loading instances times will sometimes affect your placement in google and could damage your quality score
    if ads and marketing with Adwords. Well I’m adding this RSS to my email and can look out for much more of your respective intriguing content.
    Ensure that you update this again very soon.

    Here is my blog post; cbd oil, tinyurl.com,

  5. Hi there! This is my first visit to your blog!
    We are a team of volunteers and starting a new initiative in a community in the same niche.

    Your blog provided us valuable information to
    work on. You have done a outstanding job!

    Feel free to surf to my blog – cbd oil (http://www.helios-ps.com)

  6. Heya exceptional website! Does running a blog similar to this
    take a large amount of work? I’ve absolutely
    no expertise in programming but I was hoping to start
    my own blog soon. Anyhow, if you have any
    suggestions or techniques for new blog owners please share.

    I understand this is off topic but I just had to ask. Kudos!

    Look at my web-site … cbd oil that works 2020

  7. Unquestionably believe cbd oil that works 2020 which you stated.
    Your favorite justification appeared to be on the net the simplest thing to be aware of.
    I say to you, I definitely get annoyed while people consider worries that they just don’t know about.

    You managed to hit the nail upon the top as well as defined out the whole thing without having side effect , people
    could take a signal. Will probably be back to get more.
    Thanks

댓글 남기기