Home

More twitter tomfoolery

December 16, 2017

I wrote a Go program to install blocks from a file of Twitter IDs. It’s not on github yet because my development sandbox looks more like a development catbox, and I can’t clean it up too much because the program’s running right now and Twitter’s rate limited so it’ll be a while (at 5 blocks per second, about 10 days). Recall that my goal is to completely remove fascists and griefers from my Twitter feed, and from any conversation that I happen to be in — they shouldn’t even notice the opportunity to respond, never mind wasting anyone’s time with their crap.

At least as important as the program is the list of IDs to block.
It’s 4 million lines long, sorted from most-to-least-desirable-to-block order, so this is the only way to share it. I did some by-hand sampling, and the first 10% really are notably more obnoxious than the last 10%, so I may not run this all the way to completion.

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/url"
	"os"
	"regexp"
	"strconv"
	"strings"
	"time"

	"github.com/BurntSushi/toml"
	"github.com/ChimeraCoder/anaconda"
)

var digits = regexp.MustCompile("[0-9]+")
var runtime = time.Now().Format(time.RFC3339)

type ConsumerAndAppKeysAndSecrets struct {
	ConKey, ConSecret, AppToken, AppSecret string
}

var caksFile = ".twitter/ids"

/* The .twitter/ids file contains four lines with string values
   obtained from the twitter developer api.

ConKey = "..."
ConSecret = "..."
AppToken = "..."
AppSecret = "..."

   For an App token and secret, you need to create an app here: https://apps.twitter.com/app/new
   This will then give you the option to create a consumer key and secret.
   (This useful information cribbed from
   https://stackoverflow.com/questions/1808855/getting-new-twitter-api-consumer-and-secret-keys )

*/

func main() {
	caks := &ConsumerAndAppKeysAndSecrets{}
	blob, err := ioutil.ReadFile(caksFile)
	if err != nil {
		fmt.Printf("There was an error opening or reading file %s: %v\n", caksFile, err)
		os.Exit(1)
		return
	}

	err = toml.Unmarshal(blob, caks)
	if err != nil {
		fmt.Printf("There was an error unmarshalling contents of %s: %v\n", caksFile, err)
		os.Exit(1)
		return
	}

	anaconda.SetConsumerKey(caks.ConKey)
	anaconda.SetConsumerSecret(caks.ConSecret)
	api := anaconda.NewTwitterApi(caks.AppToken, caks.AppSecret)
	fmt.Println("Credentials = ", *api.Credentials)

	a := os.Args
	if len(a)  0 && i%1000 == 0 {
			flush(users, i)
			users = users[:0]
		}
		user, err := api.BlockUserId(int64(u), url.Values{})
		if err != nil {
			errst := err.Error()
			if !strings.Contains(errst, "User not found.") {
				fmt.Printf("i=%d, u=%d, err=%v\n", i, u, err)
				flush(users, i)
				os.Exit(1)
			}
			fmt.Print("X")
		} else {
			fmt.Print(".")
		}
		users = append(users, user)
	}

	flush(users, len(wlids))

}

type tomlWantsStruct struct {
	users []anaconda.User
}

func flush(users []anaconda.User, ending int) {
	buf := new(bytes.Buffer)
	if err := json.NewEncoder(buf).Encode(users); err != nil {
		fmt.Printf("There was an error encoding users: %v\n", err)
		os.Exit(1)
	}
	fname := fmt.Sprintf("Blocked-%s-%08d", runtime, ending)
	err := ioutil.WriteFile(fname, buf.Bytes(), 0666)
	if err != nil {
		fmt.Printf("There was an error writing %v: %v\n", fname, err)
		os.Exit(1)
	}
}

func readFileAsUint64s(filename string) (uids []uint64, err error) {
	var b []byte
	b, err = ioutil.ReadFile(filename)
	if err != nil {
		return
	}
	bids := bytes.Split(b, []byte("\n"))
	uids = make([]uint64, 0)
	for i, bid := range bids {
		s := digits.FindString(string(bid))
		if s == "" {
			continue
		}
		var uid uint64
		uid, err = strconv.ParseUint(s, 10, 64)
		if err != nil {
			fmt.Printf("Failure to parse line %d of %s\n", i, filename)
			return
		}
		uids = append(uids, uid)
	}
	return
}

2 Responses to “More twitter tomfoolery”

  1. Ray Jones Says:

    Cool. I’m curious how you built the list?

    Like

    • dr2chase Says:

      Somewhat described here.

      Or TLDR, start with existing block list, extract all users with damning keywords in their name, call this the “vile” set. I used a somewhat larger list of damning keywords than is described in the linked post above — I’m sure you can imagine some likely words, and I also did some grep-ology to ensure that I was not getting obviously stupid false positives.

      The list to block is “followers(vile) – friends(friends(me))”, except that there need to be some tweaks applied to remove users with very large numbers of followers and/or friends. Then I sort the block list by the number of vile users that they follow; those who follow the most (thousands!) are most blockable. #48084 is the first that follows only 100 in the vile set. #200000 follows 15, #300000 follows 8.

      I get the profile back for each person I block, I’m binning them by 1000s, I figure I can do some post-processing to see if this can be refined somewhat, though I may not bother. From sampling, the people who follow lower numbers of vile people really are less obviously awful. I’ve already had one “hit” today, where someone I follow quote-replied a jerk that I had blocked within the previous hour.

      It’s somewhat easier to be casual about this because I am only making “policy” for myself and like-minded nerds with time on their hands, though I am a little curious what happens to Twitter with a million-long block list.

      Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: