Building the Writer’s Knowledge Base—by Mike Fleming

WkbBadgeHannibal from the A-Team always loved it when a plan came together. Unfortunately, the Writer’s Knowledge Base (WKB) didn’t emerge from some well thought out plan. While I’m not a believer in destiny I’ll admit that sometimes it does seem like a real force and the WKB could be an example. In this post I’ll describe how the WKB came to be, how it works, and why I’m doing it.

When I started following writers on Twitter I quickly realized that the excellent links they posted had the lifespan of a gnat. Actually, gnats live a lot longer. It seemed like a shame that the links had such a short shelf life. The actual page at the other end of the link was still there, of course, but finding it is a lot trickier when you don’t have a human curator separating the good from the bad.

While writing this post I dug through my notes to find what I wrote about the idea I had for fixing the problem. I found a less-than-eloquent entry on September 30, 2010, that says:

“Monitor writers’ tweets for links to writerly subjects especially on the craft of writing. Then, user could search for “characterization” and get links to all kinds of articles.”

While those two sentences clearly foreshadow the WKB as it is today, back in September it was just another idea in a bucket full of them. While I suspected it was a good idea I decided to continue focusing on Hiveword which is the fiction organizer I’m developing. In fact, the idea itself was intended to be part of Hiveword at some point. That’s the context I was in at the time.

Now keep in mind that @elizabethscraig is one of the Twitterers I was following and while she is not the only one to post links I think we can all agree that she is by far the most prolific one. So imagine my surprise when I saw her post on December 13th where she was exasperated about the difficulties of making all of those great links findable.

Well.

The problem was she had content and no technology and I had technology and no content. Isn’t that how Reese’s peanut butter cups were born?

This smacks of destiny, I thought. So, I slept on it and on the next day sent Elizabeth an email outlining my proposed solution. After running a background check on me she decided that together we could provide a compelling free service to writers everywhere. Bloggers would benefit, too, since they would have another source of traffic. There was little downside.

With Elizabeth on board I set off to work. From concept to implementation it took under a month to do on a part-time basis since I have a day job. Part of the reason it was so fast was that I was able to leverage the platform I already had for Hiveword. Another reason is that I had an appendectomy in early January and the doctor said I should stay home for a week. How convenient. 40+ hours of work on the WKB. w00t!

Of course, telling you how it works would spoil some of the magic, no? I think you’ll find that it’s actually fairly mundane. But if you insist…

The WKB automatically checks Elizabeth’s Twitter feed once an hour, pulls any new tweets since the last time, and stores them in a holding area in the database. Each day I manually process each link by copying and pasting the article content into the search engine component. The search engine indexes the content and makes it fast to search. That’s where all the magic is, of course.

You might be wondering why I do the manual part when I could have the computer do it. I’ll tell you, that’s a mighty fine question given the number of links Elizabeth tweets! The answer is simple, though, and it’s about search quality. If I index the entire article the search engine component will consider the whole page including header, footer, sidebars, ads, comments, etc. That can obviously throw off the relevance score when you do a search. If bloggers would agree on a standard way of marking content I could pull it automatically but there’s not enough consistency for me to do that.

Then there was the fact that Elizabeth already had approximately 5,500 links on her Twitterific pages. Those links were just sitting there daring me to get them indexed. I’m pleased to say that I corralled those rascals but I didn’t process them manually, of course. Rather, the entire page of each article was indexed which, unfortunately, has the drawbacks mentioned above. That’s why you’ll sometimes see strange snippets under a result. Sorry about that. However, there are now more than 6,000 articles in the WKB for you to learn from and enjoy.

Finding articles is rather easy because the interface is intentionally simple a la Google. Searching is an obvious way to find articles but you can find plenty of gems by trying the Random or Popular links. Random is self-explanatory and Popular would perhaps be better named “Top 100” since that’s what it’s really showing. Give them a try if you haven’t already; I think you’ll be pleasantly surprised.

You might also be wondering why I’m doing this. There are actually a bunch of reasons. For example, as a programmer I’ve benefited greatly from the work of others on the Internet who gave freely of their time and skill and I’ve wanted to contribute something back for a while now but hadn’t been able to hit on the right thing. I always assumed it would be something for programmers but giving something to the writing community is like paying it forward. That works for me.

Also, the WKB amplifies the work that Elizabeth is doing for writers so that’s a win, too. As mentioned earlier bloggers will get more recognition and traffic and users of the WKB will hopefully learn something from their time spent using it. That means four distinct parties can benefit from the WKB — how great is that?

I’m elated by the reception the WKB has gotten from the writing community and I’m pleased that so many get value from it. I enjoyed creating the WKB and of course it wouldn’t be as useful as it is without Elizabeth’s help. She does a huge amount of work digging up the content in the first place.

That said, the WKB is not done. Would you believe I have a bucket full of ideas for it? Stay tuned!

———————–

Mike Fleming is a software engineer who can’t seem to get enough of his craft. Give him something to do by suggesting some features for the WKB. He also maintains the WKB’s Facebook page which he considers a place for insiders to stay informed about WKB news and tips. You can also sign up for the Hiveword email list if you want to be notified when the fiction organizer is ready.

Elizabeth Spann Craig

View posts by Elizabeth Spann Craig
Elizabeth writes the Memphis Barbeque series (as Riley Adams) and the Southern Quilting mysteries for Penguin and writes the Myrtle Clover series for Midnight Ink and independently. She also has a blog, which was named by Writer’s Digest as one of the 101 Best Websites for Writers. There she posts on the writing craft, finding inspiration in everyday life, and fitting writing into a busy schedule.

19 Comments

  1. Charmaine ClancyFebruary 14, 2011

    This sounds amazing, I can’t wait to check it out. Big pat on the back for a great partnership!

  2. Rebecca BradleyFebruary 14, 2011

    I’m with Charmaine. It sounds great and I can’t wait!

  3. Margot KinbergFebruary 14, 2011

    Elizabeth – Thanks so much for hosting Mike.

    Mike – Thanks for explaining your side of how this whole partnership came about. And I’m with Elizabeth; I love the Reese’s analogy! Thanks also for all of the work that you are doing with the Writer’s Knowledge Base :-).

  4. Terry OdellFebruary 14, 2011

    Well Done! Kudos! It’s already so far beyond my comprehension, I can’t say more. Except Happy Valentine’s Day.

    Terry
    Terry’s Place
    Romance with a Twist–of Mystery

  5. Elizabeth Spann Craig/Riley AdamsFebruary 14, 2011

    I love Mike’s Reese’s analogy because that’s exactly what it was like. I was getting very frustrated by not finding a good solution for storing my writing links. Plus, I was determined that the links had to be searchable. We all don’t need the same info at the same time…sometimes I need brainstorming ideas, sometimes I need editing tips. Sometimes I need help with promo. It’s all cyclical with publishing.

    Mike’s solution made so much sense. Actually, it really blew my mind–we can do that? We can make a separate search engine? It was perfect.

    Thanks so much, Mike!

  6. Carol KilgoreFebruary 14, 2011

    This is a wonderful thing both of you are doing. Thank you!

  7. Mike FlemingFebruary 14, 2011

    Elizabeth — You’re welcome. Thanks for finding all that great content!

    Everyone — The Facebook page (http://www.facebook.com/pages/WritersKB/128813660518131) is having problems at the moment but I posted the original mockup of the search page. It goes nicely with this post if only FB would work. ;-)

  8. The Old SillyFebruary 14, 2011

    WKB sounds great – I’ll be checking it out …

    BTW, that line … “the lifespan of a gnat” – lol, love that one! ;-)

    Marvin D Wilson

  9. Cleo CoyleFebruary 14, 2011

    Mike and Elizabeth – This is off the hook brilliant. I just signed up for your Writer’s Knowledge Base newsletter via your facebook page. Cheers and thank you for this wonderful resource for writers (pro or aspiring).

    Cheers to you both!
    ~ Cleo Coffeehouse Mystery.com
    …”
    Cleo Coyle on Twitter

  10. TK RichardsonFebruary 14, 2011

    It’s an excellent resource. You’ve both done such a great job. :) And, yes, the Reese’s analogy is perfect. Thanks for all the hard work you’re doing for other writers.

  11. L. Diane WolfeFebruary 14, 2011

    Wow, so much work! We appreciate it though.

  12. BluestockingFebruary 14, 2011

    Mike, thanks so much for making Elizabeth’s hard work more visible. The WKB is wonderful! And of course, I want to know more about your future plans for it.

  13. Jane Kennedy SuttonFebruary 14, 2011

    I think it is a fantastic and helpful site. Thank you Elizabeth and Mike!

  14. Hart JohnsonFebruary 14, 2011

    Well Mike, I just REALLY want to thank you for this! I’ve shared the link a ton of places, as you’re right–there is a ton of good info, but finding it can be challenging.

    (Elizabeth, you already KNOW I think you’re a superstar)

  15. Alex J. CavanaughFebruary 14, 2011

    Thanks for putting so much into the Writer’s Knowledge Base, Mike!

  16. Elizabeth Spann Craig/Riley AdamsFebruary 14, 2011

    Thanks for coming by today, everyone! I’m glad that everyone is as excited as I am by this. I really feel like it’s such a great way to share resources…and I’m looking forward to adding links every day. I’m learning a ton by just tweeting them. :)

  17. Mike FlemingFebruary 15, 2011

    Sorry it took me so long to respond…

    Thank you, everyone, for the kind words. It’s a pleasure doing something like this when people actually use it.

    Old Silly — LOL. Just so you know, a gnat’s lifespan is 2-4 months. I do my research. ;-)

    Cleo — May we use “off the hook brilliant” as a testimonial? :-)

    Bluestocking — I may start listing some ideas on the Facebook page if you want to follow along there. Actually, I’d like to get everyone involved in the future direction so feel free to post your feature ideas on the FB page.

    Finally, thanks to Elizabeth for letting me do a guest post.

  18. Kathleen ErnstFebruary 15, 2011

    What an interesting idea! Kudos for all the thought, work, and energy.

  19. JanetFebruary 17, 2011

    This sounds great! I am going to check it out. Thanks.

Comments are closed.

Scroll to top